It is common for a computer processor and associated operating system to have two different levels of resources and protection. One level, referred to as a non-privileged mode or user mode, is used by various operating system components, application programs, and other so-called "user" processes or programs. At this level, an execution thread is prevented by the operating system and by the computer processor from performing certain security-critical operations. The thread is also prevented from directly accessing many system resources. The purpose of the non-privileged execution mode is to isolate a user process as much as possible so that it cannot interfere with other user processes or with operating system functions. While a user process may itself crash, it should not be able to crash other programs or the operating system.
The other level of execution is referred to as privileged mode, system mode, or kernel mode. Critical operating system components are implemented in kernel mode-kernel-mode components are responsible for things like virtual memory management, responding to interrupts and exceptions, scheduling execution threads, synchronizing the activities of multiple processors, and other critical or sensitive functions. Such components, which execute from system mode, are generally referred to collectively as "the kernel."
A typical approach to managing computer system memory is to allocate some portion of the memory to the operating system or kernel and another portion for application or user processes. The kernel typically runs in a dedicated or private virtual address space. The kernel or system virtual address space is protected, by processor hardware, from being accessed by any execution thread that is not in the privileged execution mode. Therefore, in practice, the kernel address consists of a range of virtual addresses that cannot be directly accessed by user processes. Current microprocessors have a 4 gigabyte virtual address space, which is typically divided for use by the kernel and by a user process, respectively. For example, the upper 2 gigabytes of these addresses might be reserved for the kernel, while the lower 2 gigabytes would be available for a user process. Since only one user process typically executes at any given time, the portion of the system memory allocated for a user process is entirely used by the current executing process. Each user process would than have its own 2 gigabyte virtual address space. A user process may conventionally access only the system memory allocated to the user process. The kernel may access both the kernel system memory and the user process system memory.
The kernel is responsible for supervising the virtual memory system in most computer systems. With virtual memory, a process is assigned its own virtual address space, which is not available to other processes. Through its virtual memory, a process has a logical view of memory that does not correspond to the actual layout of physical memory. Each time a process uses a virtual memory address, the virtual memory system translates it into a physical address using a virtual-to-physical address mapping contained in some type of look-up structure and address mapping database.
FIG. 1 illustrates the mapping of a virtual memory space 10 to physical memory 12. The virtual memory space is divided into blocks 14 of equal size called pages. Physical memory is divided into blocks 16 called page frames, which are used to hold pages. Data structures 18, usually including what is known as a "translation lookaside buffer" (TLB), are used to store the mappings between virtual memory pages 14 and physical memory page frames 16. Each time a process uses a virtual memory address, the virtual memory system translates it into a physical address using a virtual-to-physical address mapping contained in data structures 18. In many systems, memory paging is used to increase the effective size of physical memory. In paging systems, physical page frames can be swapped to a hard disk, and replaced in physical memory as they arc needed.
Notice that not all pages of the virtual address space are actually mapped to physical page frames-only those pages actually used by the associated process are mapped or "allocated." Generally, a process is only allowed to access these allocated virtual memory addresses. Accessing other addresses generates a memory fault or exception that in some cases causes the offending process to be terminated. For purposes of this description virtual addresses that have not been allocated-that have not been mapped to physical memory-are referred to as invalid virtual memory addresses. The term invalid as used herein does not refer to virtual memory addresses whose corresponding physical memory has been paged to disk.
It is often necessary for a user process to invoke system or kernel functions. When a user process calls a system function, the computer processor traps the call and switches the current execution thread from user mode to kernel mode. The kernel takes control of the thread, validates the arguments received from the thread and then executes the designated system function. The operating system switches the thread back to user mode before returning control to the user process. In this way, only system functions are allowed to execute from the privileged kernel mode. This protects the kernel and its data from perusal and modification by user processes.
It is often necessary for kernel functions to accept data from and to provide data to user processes. This is effectively accomplished using memory buffers or regions, located in the virtual address space of the user. The locations of such buffers are specified to kernel functions as pointer arguments in system calls. Each pointer contains a virtual memory address.
However, to protect its integrity the kernel must guard itself against faulty pointers supplied by user processes. For example, a user process might mistakenly or maliciously supply a pointer to a protected kernel address, thus potentially causing the kernel to write over itself. This must be prevented. As another example, a user process might mistakenly supply an invalid pointer-one containing a virtual memory address that has not been allocated to physical memory. In this situation, the kernel must be able to handle the resulting memory fault gracefully, without terminating.
In prior art Unix systems, protection against such faulty pointers can be implemented using copying and copyout functions. Prior to using a user-supplied pointer, the kernel is programmed to make a copy of the data referenced by the pointer, using the Unix-supplied copying and copyout functions. These functions perform bounds checking to ensure that specified pointers are not in the kernel's address space, and also detect invalid pointers. The function returns an error code if there is any problem with the pointer. While this approach works, it is very inefficient since actual data copies are made.
In other operating systems, bounds checking and "try-except" procedures are implemented in the kernel before actually using any user-supplied pointer. However, this requires specific procedures to be implemented at many places in the kernel, it is inefficient and prone to errors.
A third prior art approach is to examine currently existing virtual-tophysical address mappings to determine if a pointer is valid. However, this approach is extremely inefficient and does not work in multi-tasking or multi-processor environments. In such an environment, a process might check and determine that a particular pointer is valid, be preempted, and then resume execution without realizing that the pointer is no longer valid.
The inventors have discovered a very effective way to protect kernel functions against faulty pointers, without requiring any special measures within the individual kernel functions themselves.